Methods And Apparatuses For Controlling Thread Contention

ABSTRACT

An apparatus comprises a plurality of cores and a controller coupled to the cores. The controller is to lower an operating point of a first core if a first number based on processor clock cycles per instruction (CPI) associated with a second core is higher than a first threshold. The controller is operable to increase the operating point of the first core if the first number is lower than a second threshold.

This application is a continuation of U.S. patent application Ser. No.12/414,504, filed Mar. 30, 2009, the content of which is herebyincorporated by reference.

FIELD OF THE INVENTION

Embodiments of the invention relate to the field of computer systems;more particularly, to controlling thread contention in a computersystem.

BACKGROUND OF THE INVENTION

Advances in semiconductor processing and logic design have permitted anincrease in the amount of logic on integrated circuit devices. As aresult, computer system configurations have evolved from a single ormultiple integrated circuits in a system to multiple cores and multiplelogical processors present on individual integrated circuits. Aprocessor or integrated circuit typically comprises a single processordie that has any number of processing resources, such as cores, threads,or logical processors.

In a processor with multiple threads, the behavior of one threadpotentially affects the behavior of another thread executing thereonbecause of sharing of resources, such as, for example, caches, memory,and power.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be understood more fully fromthe detailed description given below and from the accompanying drawingsof various embodiments of the invention, which, however, should not betaken to limit the invention to the specific embodiments, but are forexplanation and understanding only.

FIG. 1 shows an embodiment of a computer system including an apparatusto control resource contention to shared resources, based on performancefeedback.

FIG. 2 is a flow diagram of one embodiment of a process to controlresource contention to shared resources, based on performance feedback.

FIG. 3 is a flow diagram of one embodiment of a process to regulatepower or clock throttling.

FIG. 4 is a flow diagram of one embodiment of a process to regulateclock throttling based on outputs from a controller.

FIG. 5 illustrates a computer system for use with one embodiment of thepresent invention.

FIG. 6 illustrates a point-to-point computer system for use with oneembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, numerous details are set forth to providea more thorough explanation of embodiments of the present invention. Itwill be apparent, however, to one skilled in the art, that embodimentsof the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form, rather than in detail, in order to avoidobscuring embodiments of the present invention.

In other instances, well-known components or methods, such as, forexample, microprocessor architecture, virtual machine monitor, powercontrol, clock gating, and operational details of known logic, have notbeen described in detail in order to avoid unnecessarily obscuring thepresent invention.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of present invention also relate to apparatuses forperforming the operations herein. Some apparatuses may be speciallyconstructed for the required purposes, or it may comprise a generalpurpose computer selectively activated or reconfigured by a computerprogram stored in the computer. Such a computer program may be stored ina computer readable storage medium, such as, but not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs, DVD-ROMs,and magnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, NVRAMs, magnetic or optical cards, orany type of media suitable for storing electronic instructions, and eachcoupled to a computer system bus.

The method and apparatus described herein are for controlling resourcecontention by regulating clock throttling and power. Specifically,regulating clock throttling and power is discussed in reference tomulti-core processor computer systems. However, the methods andapparatus for controlling resource contention are not so limited, asthey may be implemented on or in association with any integrated circuitdevice or system, such as cell phones, personal digital assistants,embedded controllers, mobile platforms, desktop platforms, and serverplatforms, as well as in conjunction with any type of processingelement, such as a core, a hardware thread, a software thread, or alogical processor, an accelerator core or other processing resource. Inaddition, controlling resource contention may take place in anyhardware/software environment, such as an operating system or ahypervisor executing on hardware.

OVERVIEW

Embodiments of a method and apparatus for controlling resourcecontention to shared resources by regulating clock throttling and powerof a processor are described. In one embodiment, the processor is amulti-core processor comprising two or more cores. In one embodiment, acontroller regulates the cores to increase performance of a hardwarethread in conjunction with a feedback mechanism including aproportional-integral-derivative controller (PID controller).

FIG. 1 shows an embodiment of a computer system including an apparatusto control resource contention to shared resources. Many relatedcomponents such as buses and peripherals have not been shown to avoidobscuring the invention. Referring to FIG. 1, the computer systemincludes power/performance setting logic 110, proportional-integralcontroller (PI controller) 120, monitor logic 160, decision logic 127,operating point control logic 128, processor 130, cache 150, and memory(not shown). In one embodiment, PI controller 120, decision logic 127,operating point control 128, cache 150, or any combination thereof isintegrated in processor 130.

In one embodiment, a computer system includes input/output (I/O) buffersto transmit and receive signals via interconnect (e.g., 111, 124, 136).Examples of the interconnect include a Gunning Transceiver Logic (GTL)bus, a GTL+ bus, a double data rate (DDR) bus, a pumped bus, adifferential bus, a cache coherent bus, a point-to-point bus, amulti-drop bus or other known interconnect implementing any known busprotocol.

In one embodiment, processor 130 includes multiple processing elements,such as processing elements 131-134. A processing element comprises athread, a process, a context, a logical processor, a hardware thread, acore, an accelerator core or any processing element, which shares accessto other shared resources of processor 130, such as, for example,reservation units, execution units, higher level caches, memory, etc. Inone embodiment, a processing element is a thread unit, i.e. an elementwhich is capable of having instructions independently scheduled forexecution by a software thread. In one embodiment, a physical processoris an integrated circuit, which includes any number of other processingelements, such as cores or hardware threads.

In one embodiment, a core is logic located on an integrated circuitcapable of maintaining an independent architectural state with respectto another core. Each independently maintained architectural state isassociated with at least some dedicated execution resources. In oneembodiment, a hardware thread is logic located on an integrated circuitcapable of maintaining an independent architectural state with respectto another hardware thread. Each independently maintained hardwarethread shares access to execution resources. In some embodiments, a coreand a hardware thread are used interchangeably. In one embodiment, acore or a hardware thread is also referred to as a processing element.

In one embodiment, a hardware thread, a core, or a processing element isviewed by an operating system or management software as an individuallogical processor. Software programs are able to individually scheduleoperations on each logical processor. Additionally, in some embodiments,each core includes multiple hardware threads for executing multiplesoftware threads.

In one embodiment, a hypervisor (not shown) provides an interfacebetween software (e.g., virtual machines) and hardware resource (e.g.,processor 130). In one embodiment, a hypervisor abstracts hardware sothat multiple virtual machines run independently in parallel. In oneembodiment, a virtual machine provides a software execution environmentfor a program, such as, for example, a task, a user-level application,guest software, an operating system, another virtual machine, a virtualmachine monitor, other executable code, or any combination thereof. Inone embodiment, a hypervisor allocates hardware resources (e.g., a core,a hardware thread, a processing element) to different programs.

In one embodiment, power/performance goal 100 is a user configurablesetting. In other embodiment, power/performance goal 100 is determinedbased on a power saving profile, a user setting, an operating system, asystem application, a user application, or the like. In one embodiment,power/performance setting logic 110 receives information frompower/performance goal 100.

In one embodiment, power/performance setting logic 110 stores targetvalues of power consumption of processor 130 or a system in whichprocessor 130 is located. In one embodiment, the computer system isreferred to herein as a platform.

In one embodiment, power/performance setting logic 110 stores a targetvalue of cycles per instruction (CPI) associated with a programexecuting on a processing element. In one embodiment, the program is ahigh priority program. The CPI value is used as a set point for PIcontroller 120. In one embodiment, information about misses perinstruction (MPI), cache line fills, cache line evictions, etc. is usedin conjunction with CPI. It will be appreciated by those of ordinaryskill that other metrics may be used as performance data with respect toa processing element.

In one embodiment, the computer system includes a number of sharedresources for which the hardware threads contend. In one embodiment, theshared resources include cache, a translation lookaside buffer (TLB),memory, and power. In one embodiment, a computer system is required torun under specific power constraints.

In one embodiment, low priority programs are of lesser importance to anend-user. In another embodiment, a low priority program is a programthat is not able to fully benefit from the maximum core CPU potentialperformance because the program involves a lot of memory requests, orintensive I/O requests which causes long waiting time.

In one embodiment, a low priority program competes for a same set ofshared resources with a high priority program in a multi-core processorsystem. In one embodiment, behavior of one core (executing a lowpriority program) creates unfairness in the usage of shared resourcesand pipelines. As a result, an unpredictable variability in performancefrom the unbalanced usage of shared resources occurs. In one embodiment,it will be beneficial to control shared resource contention byregulating a core associated with a low priority program, such as, forexample, reducing a power state of the core, reducing a clock frequencyby clock throttling, or both. For example, a background application,such as virus scan, executes on a first core utilizing enough sharedresources to adversely affect the performance of a second core, which isexecuting a foreground application (a high priority program).

In one embodiment, monitor logic 160 receives or determines data, suchas, for example, cache occupancy, cache line fills, cache lineevictions, memory bandwidth 151, power consumption 152, memory capacity,and input/output requests, which are associated with usage of variousshared resources. In one embodiment, the data are associated with anapplication, a software thread, a hardware thread, a platform, orcombinations thereof. In one embodiment, monitor logic 160 also receiveCPI information of each core (e.g., cores 131-134) via interface 136.

In one embodiment, PI controller 120 is coupled to power/performancesetting logic 110 via 111 to receive a set point (e.g., powerconsumption target, CPI target value). In one embodiment, the set pointis for a CPI value of a core associated with a high priorityapplication. In one embodiment, the set point is for a power consumptionvalue of processor 130.

In one embodiment, PI controller 120 also receives feedback data (e.g.,CPI of each core from monitor logic 160, power consumption value frommonitor logic 160). In other embodiment, such information is receivedfrom processor 130 and a power regulator directly. In one embodiment,monitor logic 160 is a part of a processor performance monitoringcomponents, an integrated part of platform components, or both.Controlling resource contention in conjunction with PI controller 120will be described in further detail below with additional references tothe remaining figures.

In one embodiment, PI controller 120 is configured by changingparameters such as, an integral gain (Ki) 122 and a proportional gain(Kp) 123. In one embodiment, PI controller 120 further comprises aderivative gain (Kd). In one embodiment, Kp is set to 2.0, Ki is set to0.3, and Kd is set to 0. In one embodiment, PI controller 120 is used toreduce an overshoot and ringing effect, such that the regulatingmechanism does not react too quickly to feedback of performance data. Inother words, PI controller 120 provides a smoother output response thansimple rule-based determination. In one embodiment, parameters (e.g.,Kp, Ki, and Kd) are adjusted to improve the response of output fromcontroller 120. It will be appreciated by those skilled in the art thatthese parameters may be scaled up or down to adjust a degree ofaggressiveness of a control mechanism.

In one embodiment, decision logic 127 is coupled to receive an outputfrom PI controller 120 via interconnect 124. Decision logic 127determines whether to increase, to decrease, or to maintain theenforcement of power/clock throttling based on the output from PIcontroller 120 and some threshold values. In one embodiment, increasingor decreasing enforcement is performed by regulating power of processor130, by regulating clock throttling of the cores (e.g., cores 131-134),or both. Operations of enforcement will be described in further detailbelow with respect to operating point control 128.

In one embodiment, the threshold values are associated with settings inpower/performance setting logic 110 (e.g., user configurable, preset bysystems, etc.). In one embodiment, no action is required if the outputfrom PI controller 120 is within a range (a lower bound and an upperbound, e.g., −0.5% and 0.5% respectively). Determination based on thesettings of threshold values will be described in further detail belowwith additional references to the remaining figures (e.g., FIG. 3).

In one embodiment, a computer system, and in particular, processor 130supports different operating points (e.g., performance states (P-states)and clock throttling states (T-states)), in accordance with AdvancedConfiguration and Power Interface (ACPI) specification (see, AdvancedConfiguration and Power Interface Specification, revision 3.0b, Oct. 10,2006). In one embodiment, C0 working state of a processor is dividedinto P-states (performance states) in which clock rate is reduced andT-states (throttling states) in which clock rate is throttled byinserting STPCLK (stop clock) signals and thus omitting duty cycles. Inone embodiment, a P-state and a T-state of processor 130 are set bychanging values of one or more model specific registers (MSRs).

In one embodiment, processor 130 support various P-States, P0 throughPn, P0 being the highest state and Pn being the lowest state. At a P0state, processor 130 runs at a highest frequency. At a Pn state,processor 130 runs at a lower frequency corresponding to a greater valueof n. In one embodiment, processor 130 is capable of operating at sixdifferent performance states (P0-P5). For example, from P0 through P5,processor 130 operates at 100%, 90%, 85%, 75%, 65%, and 50% of fullperformance respectively. In one embodiment, switching to the variousP-states is also referred to as dynamic voltage and frequency scaling(DVFS).

In one embodiment, increasing a P-state is performed by transitioningthe P-state from a lower state to a higher state (e.g., P1 to P0), whereprocessor 130 will operate at a higher frequency (thus consuming powerat a higher rate) following the change. In one embodiment, theseperformance states are only valid when processor 130 is in a power stateC0.

In one embodiment, processor 130 supports various T-States, T0 throughTn, T0 being the highest state and Tn being the lowest state. In oneembodiment, when operating at a T-state, processor 130 is forced to bein an idle state (stop performing an operation) a percentage of the dutycycles. At a T0 state, processor 130 runs at a 100% of duty cycles. At aTn state, processor 130 runs at a lower percentage of duty cyclescorresponding to a greater value of n. In one embodiment, processor 130is capable of operating at six clock throttling states (T0-T5). Forexample, from T0 through T5, processor 130 operates at 100%, 90%, 80%,70%, 60%, and 50% of all duty cycles respectively. In one embodiment,switching to different T-states is also referred to as clock modulation,frequency modulation, clock-gating, etc.

In one embodiment, increasing a T-state is performed by transitioningthe T-state of a core (e.g., cores 131-134) from a lower state to ahigher state (e.g., T1 to T0), where the core will operate at a higherpercentage of duty cycles following the change. In one embodiment, theseclock throttling states are only valid when processor 130 is in a powerstate C0.

In one embodiment, a combination of a P-state and a T-state of processor130 is referred to herein as an operating point. In one embodiment,operating point control 128 controls and manages a P-state, a T-state,or both. In one embodiment, operating point control 128 stores a currentP-state, a current T-state, or both. In one embodiment, operating pointcontrol 128 sets a next operating point (changing a P-state, a T-state,or both) based on current states and the output from decision logic 127.In one embodiment, operating point control 128 decreases an operatingpoint (lower to a different operating point) by decreasing a T-state,decreasing a P-state, or both. In one embodiment, operating pointcontrol 128 increases an operating point by increasing a T-state,increasing a P-state, or both.

In one embodiment, operating point control 128 regulates power and clockthrottling of processor 130. In one embodiment, operating point control128 receives output from decision logic 127 on whether more enforcementis required or otherwise. In one embodiment, decision logic 127 inintegrated with operating point control 128.

In one embodiment, if the output from decision logic 127 indicates moreenforcement is required, operating point control 128 decreases theT-state, as long as the T-state is not at the lowest state (e.g. T5).Otherwise, operating point control 128 decreases the P-state. In oneembodiment, operating point control 128 is programmed to use onlycertain T-states (e.g., T0-T2) instead of all T-states available.

In one embodiment, if the output from decision logic 127 indicates thatless enforcement is required, operating point control 128 increases theP-state if the P-state is at a highest state (e.g. P5). Otherwise,operating point control 128 increases T-state. In one embodiment,operating point control 128 is programmed to use only certain T-states(e.g., T0-T2) instead of all T-states available.

In one embodiment, operating point control 128 sets an operating point(a combination of P-state and T-state) based on determination fromdecision logic 127. In one embodiment, decision logic 127 furtheroperates in conjunction with information including misses perinstruction, a number of cache line fills, and a number of cache lineevictions to avoid overcorrecting by PID controller 120. For example, ifperformance does not improve after reducing a P-state (for example, acache-streaming application is the root cause of resource contention),operating point control 128 performs further enforcement by reducingT-state instead. In one embodiment, in order to manage overall powerconsumption, operating point control 128 performs P-state enforcementbecause more power saving is yielded by reducing a P-state than reducinga T-state. The operations will be described in further detail below withadditional references to the remaining figures.

In one embodiment, a computer system further includes memory (not shown)to store associations of a program and a corresponding core on which theprogram executing. In one embodiment, the memory further stores aquality of service requirement (QoS), priority information, etc.associated with each program. Operating point control 128 performsenforcement on the proper cores based in part on contents of the memory.

In one embodiment, computer system registers (not shown), accessible byan operating system, are used for configuring operating point control128, decision logic 127, monitor logic 160, and PI controller 120. Inone embodiment, PI controller 120, monitor logic 160, decision logic127, and operating point control 128 operate independently of anoperating system. In one embodiment, monitor logic 160 and decisionlogic 127 operate in conjunction with an operating system to regulatepower and clock throttling of the cores (e.g., cores 131-134).

In one embodiment, an operating system schedules time (time-slicing) todifferent applications based on their priorities. A low priority programis allocated with a shorter time-slice than a high priority program. Inone embodiment, such time-slicing is not effective in controllingresource contention if a high priority is running in parallel with otherlow priority programs in a system (with multiple processing elements).In one embodiment, the performance degradation caused by resourcecontention is mitigated by regulating the cores associated with lowpriority programs.

FIG. 2 is a flow diagram of one embodiment of a process to controlresource contention to shared resources. The process is performed byprocessing logic that may comprise hardware (circuitry, dedicated logic,etc.), software (such as is run on a general purpose computer system ora dedicated machine), or a combination of both. In one embodiment, theprocess is performed in conjunction with a PI controller (e.g., PIcontroller 120 with respect to FIG. 1). In one embodiment, the processis performed by a computer system with respect to FIG. 5.

Referring to FIG. 2, the process begins by processing logic readingconfigurations, such as, for example, performance goal in terms of CPI(processing block 200). In one embodiment, processing logic also receiveinformation including priorities of applications, power constraints,performance targets, etc. (process block 250).

In one embodiment, processing logic determines a control option (processblock 210). In one embodiment, the control option includes at least oneof control mode 0 (indicated as 261), control mode 1 (indicated as 262),control mode 2 (indicated as 263), or any combinations thereof.

In one embodiment, processing logic slows down a low priority program toincrease performance of a high priority program when operating incontrol mode 0. In one embodiment, processing logic slows down one ormore programs which are of lower priorities with respect to anothergroup of programs, when operating in control mode 0.

In one embodiment, processing logic sets a set point of a PI controller(process block 203). In one embodiment, the set point is a target forperformance (e.g., 1.2x). In one embodiment, the value is 1.0x when aprogram is executing at full performance (e.g., executing alone on theplatform without resource contention). In one embodiment, the value isset to 1.2x to indicate that 20% slow down from the full performance isacceptable. In one embodiment, a performance target is based on workloadperformance metrics, such as, for example, transactions per minute,operations per second, etc.

In one embodiment, processing logic monitors CPI of a core executing ahigh priority program (process block 204). In one embodiment, processinglogic also monitors information, such as, for example, misses perinstruction (MPI), cache line fills, cache line evictions, etc.

In one embodiment, processing logic compares data from the monitoringwith the set point (process block 205). In one embodiment, no action istaken if the performance data are within a predetermined range. In oneembodiment, if the performance data are higher than the set point, a PIcontrol mechanism generates an output based at least on the difference(error) between the set point and performance data (process block 206).In one embodiment, processing logic determines, based on the output,whether to increase a T-state (less enforcement) or to decrease aT-state (more enforcement) of cores associated with low priorityprograms (process block 207).

In one embodiment, processing logic, when operating in control mode 1,slows down a program that initiates too many memory requests which, inturn, results in long memory latency. In one embodiment, processinglogic also controls a program that causes cache-streaming. In oneembodiment, memory latency is an indication of memory bandwidthutilization. Long memory latency indicates that the memory system isoverloaded with requests from low priority programs.

In one embodiment, processing logic sets a set point of a PI controller(process block 213). In one embodiment, the set point is a memorylatency threshold value (e.g., 65 ns).

In one embodiment, processing logic monitors memory latency associatedwith a core executing a high priority program (process block 214). Inone embodiment, processing logic also monitors information, such as, forexample, misses per instruction (MPI), cache line fills, cache lineevictions, etc. In one embodiment, a high MPI or a large number of cacheline evictions indicates that some programs sharing the same memorysystem are cache streaming programs.

In one embodiment, processing logic compares data from the monitoringwith the set point (process block 215). In one embodiment, no action istaken if the memory latency is within a predetermined range. In oneembodiment, if the memory latency is higher than the set point, a PIcontrol mechanism generates an output based at least on the difference(error) between the set point and the memory latency (process block216). In one embodiment, processing logic determines, based on theoutput, whether to increase a T-state (less enforcement) or to decreasea T-state (more enforcement) of cores associated with low priorityprograms (process block 217).

In one embodiment, processing logic determines to set all coresassociated with low priority programs to operate at 75% duty cycles ifthe memory latency is higher than a set point (e.g., 65 ns). In oneembodiment, processing logic improves overall throughput by decreasing aT-state of a core that executes one or more program resulting a highMPI.

In one embodiment, processing logic, when operating in control mode 2,controls a power consumption of a system (process block 223).

In one embodiment, processing logic sets a set point of a PI controller.In one embodiment, the set point is a system power constraint (e.g., 225W). In one embodiment, processing logic monitors power consumptionassociated with the system (process block 224).

In one embodiment, processing logic compares data from the monitoringwith the set point (process block 225). In one embodiment, no action istaken if the power consumption is within a predetermined range. In oneembodiment, if the power consumption is higher than the set point, a PIcontrol mechanism generates an output based at least on the difference(error) between the set point and the power consumption (process block226). In one embodiment, processing logic determines, based on theoutput, whether to increase a T-state (less enforcement) or to decreasea T-state (more enforcement) of cores associated with low priorityprograms (process block 227).

In one embodiment, if the power consumption remains higher than the setpoint, processing logic reduces a T-state (more enforcement) of coresexecuting high priority programs. In one embodiment, processing logicalso reduces a P-state of the processor if the power consumption remainshigher than the set point.

In one embodiment, processing logic repeats the operation at eachsampling interval.

FIG. 3 is a flow diagram of one embodiment of a process to regulatepower or clock throttling. The process is performed by processing logicthat may comprise hardware (circuitry, dedicated logic, etc.), software(such as is run on a general purpose computer system or a dedicatedmachine), or a combination of both. In one embodiment, the process isperformed in conjunction with a controller (e.g., decision logic 127with respect to FIG. 1). In one embodiment, the process is performed bya computer system with respect to FIG. 5.

In one embodiment, a PI controller generates a numeric output (processblock 300). In one embodiment, processing logic determines whetheroutput from a PI controller is within a range (process block 310). Inone embodiment, no action will be taken if output from the PI controlleris within a range (process block 320). In one embodiment, the range isset to prevent the enforcement mechanism from constantly togglingbetween performance states. In one embodiment, the effect on performanceslightly lags behind an enforcement operation (a hysteresis effect). Inone embodiment, the ranges used in different control modes are shown inthe following table.

TABLE 1 Threshold values Low Mid High Mode Threshold Threshold Threshold0 −0.5 N/A 0.6 (0.5) 1 N/A 65 ns N/A 2 −10 W N/A 10 W

In one embodiment, a positive output value from a PI controllerindicates that more enforcement is required, whereas a negative outputvalue from the PI controller indicates that enforcement should bereduced.

In one embodiment, if more enforcement is required (process block 320),processing logic determines whether the current T-state is the lowestT-state (process block 330). If the current T-state is the lowestT-state (e.g., T7), processing logic determines to lower a P-state(e.g., transitioning from P0 to P1) (process block 331). Otherwise,processing logic determines to lower a T-state (e.g., transitioning fromT4 to T5) (process block 332).

In one embodiment, if less enforcement is required (process block 321),processing logic determines whether the current P-state is the highestP-state (process block 340). If the current P-state is the highestP-state (e.g., P0), processing logic determines to increase a T-state(e.g., transitioning from T5 to T4) (process block 341). Otherwise,processing logic determines to increase P-state (e.g., transitioningfrom P1 to P0) (process block 342).

In one embodiment, P-states control is only available at a socket level,such that different processing elements on a same socket receive a sameP-state setting.

In one embodiment, operating points are defined as combinations ofdifferent P-states and T-states. In one embodiment, for example, P0 andT0 are the current states. T0 through T7 are supported in the example.When more enforcement is required, processing logic selects acombination of P0/T1. Subsequently, at the next operation, processinglogic selects a combination of P0/T2 as the operating point if moreenforcement is required (based on performance data feedback frommonitoring). The process repeats until processing logic selects thecombination of P0/T7 (the lowest T-state). If more enforcement is stillrequired, processing logic selects a combination of P1/T7, followed byP2/T7 and so on.

In other embodiments, processing logic begins to lower a P-state whenthe T-state reaches at T4 (50% clock modulation) instead of T7 forefficiency reasons.

FIG. 4 is a flow diagram of one embodiment of a process to regulateclock throttling based on outputs from a controller. The process isperformed by processing logic that may comprise hardware (circuitry,dedicated logic, etc.), software (such as is run on a general purposecomputer system or a dedicated machine), or a combination of both. Inone embodiment, the process is performed in conjunction with acontroller (e.g., decision logic 127 in FIG. 1). In one embodiment, theprocess is performed in conjunction with control mode 0 or control mode1 with respect to FIG. 2.

In one embodiment, the process begin by processing logic determines anoutput from a PI controller (process block 400). In one embodiment,processing logic determines whether output from a PI controller iswithin a range (process block 410). In one embodiment, the range is setto a lower bound and an upper bound of memory latency.

In one embodiment, processing logic sets all cores associated with lowpriority programs (e.g., background applications) to operate at 75% dutycycles if the output from the PI controller is not in the range (processblock 421). In one embodiment, processing logic sets cores associatedwith low priority programs to operate at a lower duty cycles (e.g., 50%)if the output from the PI controller is out of the range.

In one embodiment, processing logic sets all cores associated with lowpriority programs to operate at 100% duty cycles (at T0) if output fromthe PI controller is back in the range (process block 422).

Embodiments of the invention may be implemented in a variety ofelectronic devices and logic circuits. Furthermore, devices or circuitsthat include embodiments of the invention may be included within avariety of computer systems. Embodiments of the invention may also beincluded in other computer system topologies and architectures.

FIG. 5, for example, illustrates a computer system in conjunction withone embodiment of the invention. Processor 705 accesses data from level1 (L1) cache memory 706, level 2 (L2) cache memory 710, and main memory715. In other embodiments of the invention, cache memory 706 may be amulti-level cache memory comprise of an L1 cache together with othermemory such as an L2 cache within a computer system memory hierarchy andcache memory 710 are the subsequent lower level cache memory such as anL3 cache or more multi-level cache. Furthermore, in other embodiments,the computer system may have cache memory 710 as a shared cache for morethan one processor core.

In one embodiment, the computer system includes quality of service (QoS)controller 750. In one embodiment, QoS controller 750 is coupled toprocessor 705 and cache memory 710. In one embodiment, QoS controller750 regulates processing elements of processor 705 to control resourcecontention to shared resources. In one embodiment, QoS controller 750includes logic such as, for example, PI controller 120, decision logic127, operating point control 128, or any combinations thereof withrespect to FIG. 1. In one embodiment, QoS controller 750 receives datafrom monitoring logic (not shown) with respect to performance of cache,power, resources, etc.

Processor 705 may have any number of processing cores. Other embodimentsof the invention, however, may be implemented within other deviceswithin the system or distributed throughout the system in hardware,software, or some combination thereof.

Main memory 715 may be implemented in various memory sources, such asdynamic random-access memory (DRAM), hard disk drive (HDD) 720, solidstate disk 725 based on NVRAM technology, or a memory source locatedremotely from the computer system via network interface 730 or viawireless interface 740 containing various storage devices andtechnologies. The cache memory may be located either within theprocessor or in close proximity to the processor, such as on theprocessor's local bus 707. Furthermore, the cache memory may containrelatively fast memory cells, such as a six-transistor (6T) cell, orother memory cell of approximately equal or faster access speed.

Other embodiments of the invention, however, may exist in othercircuits, logic units, or devices within the system of FIG. 5.Furthermore, in other embodiments of the invention may be distributedthroughout several circuits, logic units, or devices illustrated in FIG.5.

Similarly, at least one embodiment may be implemented within apoint-to-point computer system. FIG. 6, for example, illustrates acomputer system that is arranged in a point-to-point (PtP)configuration. In particular, FIG. 6 shows a system where processors,memory, and input/output devices are interconnected by a number ofpoint-to-point interfaces.

The system of FIG. 6 may also include several processors, of which onlytwo, processors 870, 880 are shown for clarity. Processors 870, 880 mayeach include a local memory controller hub (MCH) 811, 821 to connectwith memory 850, 851. Processors 870, 880 may exchange data via apoint-to-point (PtP) interface 853 using PtP interface circuits 812,822. Processors 870, 880 may each exchange data with a chipset 890 viaindividual PtP interfaces 830, 831 using point to point interfacecircuits 813, 823, 860, 861. Chipset 890 may also exchange data with ahigh-performance graphics circuit 852 via a high-performance graphicsinterface 862. Embodiments of the invention may be coupled to computerbus (834 or 835), or within chipset 890, or coupled to data storage 875,or coupled to memory 850 of FIG. 6.

Other embodiments of the invention, however, may exist in othercircuits, logic units, or devices within the system of FIG. 6.Furthermore, in other embodiments of the invention may be distributedthroughout several circuits, logic units, or devices illustrated in FIG.6.

The invention is not limited to the embodiments described, but can bepracticed with modification and alteration within the spirit and scopeof the appended claims. For example, it should be appreciated that thepresent invention is applicable for use with all types of semiconductorintegrated circuit (“IC”) chips. Examples of these IC chips include butare not limited to processors, controllers, chipset components,programmable logic arrays (PLA), memory chips, network chips, or thelike. Moreover, it should be appreciated that exemplarysizes/models/values/ranges may have been given, although embodiments ofthe present invention are not limited to the same. As manufacturingtechniques (e.g., photolithography) mature over time, it is expectedthat devices of smaller size could be manufactured.

Whereas many alterations and modifications of the embodiment of thepresent invention will no doubt become apparent to a person of ordinaryskill in the art after having read the foregoing description, it is tobe understood that any particular embodiment shown and described by wayof illustration is in no way intended to be considered limiting.Therefore, references to details of various embodiments are not intendedto limit the scope of the claims which in themselves recite only thosefeatures regarded as essential to the invention.

1. An apparatus comprising: a plurality of processing elements; and acontroller coupled to the plurality of processing elements to lower anoperating point of a first processing element if a first number based oncycles per instruction (CPI) associated with a second processing elementis higher than a first threshold.
 2. The apparatus of claim 1, whereinthe controller is operable to increase the operating point of the firstprocessing element if the first number is lower than a second threshold.3. The apparatus of claim 1, wherein the controller is operable to lowerthe operating point of the first processing element if a second numberbased on memory latency is higher than a second threshold.
 4. Theapparatus of claim 1, wherein the controller is operable to lower theoperating point of the first processing element if a second number basedon a system power value is higher than a second threshold.
 5. Theapparatus of claim 1, wherein the controller is operable to determine,based at least in part on a current operating point of the firstprocessing element, whether to regulate clock throttling of the firstprocessing element or to regulate power of the plurality of processingelements.
 6. The apparatus of claim 1, further comprising aproportional-integral controller (PI controller) to reduce a differencebetween the CPI and a set point, wherein the first number is an outputfrom the PI controller.
 7. The apparatus of claim 1, wherein the firstcontroller is operable to let the operating point remains unchanged ifthe first number is within a range.
 8. The apparatus of claim 1, furthercomprising monitoring logic to determine miss-per-instruction (MPI) anda number of cache line evictions.
 9. The apparatus of claim 1, whereinpriority of a first program executing on the first processing element islower than priority of a second program executing on the secondprocessing element.